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A complete written statement as to the substance of any face-to-face, video conference, or telephone interview with regard to an application must be made of record in the 
application whether or not an agreement with the examiner was reached at the interview. 

Title 37 Code of Federal Regulations (CFR) § 1.133 Interviews 
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substance of an interview is completely recorded in an Examiners Amendment, no separate Interview Summary Record is required. 

The Interview Summary Form shad be given an appropriate Paper No., placed in the right hand portion of the file, and listed on the 
"Contents" section of the file wrapper. In a personal interview, a duplicate of the Form is given to the applicant (or attorney or agent) at the 
conclusion of the interview. In the case of a telephone or video-conference interview, the copy is mailed to the applicant's correspondence address 
either with or prior to the next official communication. If additional correspondence from the examiner is not likely before an allowance or if other 
circumstances dictate, the Form should be mailed promptly after the interview rather than with the next official communication. 
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- Type of interview (telephonic, video-conference, or personal) 

- Name of participant(s) (applicant, attorney or agent, examiner, other PTO personnel, etc.) 
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- An identification of the specific prior art discussed 

- An indication whether an agreement was reached and if so, a description of the general nature of the agreement (may be by 
attachment of a copy of amendments or claims agreed as being allowable). Note: Agreement as to allowability is tentative and does 
not restrict further action by the examiner to the contrary. 

- The signature of the examiner who conducted the interview (if Form is not an attachment to a signed Office action) 

It is desirable that the examiner orally remind the applicant of his or her obligation to record the substance of the interview of each case. It 
should be noted, however, that the Interview Summary Form will not normally be considered a complete and proper recordation of the interview 
unless it includes, or is supplemented by the applicant or the examiner to include, all of the applicable items required below concerning the 
substance of the interview. 

A complete and proper recordation of the substance of any interview should include at least the following applicable items: 

1 ) A brief description of the nature of any exhibit shown or any demonstration conducted, 

2) an identification of the claims discussed, 

3) an identification of the specific prior art discussed, 

4) an identification of the principal proposed amendments of a substantive nature discussed, unless these are already described on the 
Interview Summary Form completed by the Examiner, 

5) a brief identification of the general thrust of the principal arguments presented to the examiner, 

(The identification of arguments need not be lengthy or elaborate. A verbatim or highly detailed description of the arguments is not 
required. The identification of the arguments is sufficient if the general nature or thrust of the principal arguments made to the 
examiner can be understood in the context of the application file. Of course, the applicant may desire to emphasize and fully 
describe those arguments which he or she feels were or might be persuasive to the examiner.) 

6) a general indication of any other pertinent matters discussed, and 

7) if appropriate, the general results or outcome of the interview unless already described in the Interview Summary Form completed by 
the examiner. 

Examiners are expected to carefully review the applicant's record of the substance of an interview. If the record is not complete and 
accurate, the examiner will give the applicant an extendable one month time period to correct the record. 

Examiner to Check for Accuracy 

If the claims are allowable for other reasons of record, the examiner should send a letter setting forth the examiner's version of the 
statement attributed to him or her. If the record is complete and accurate, the examiner should place the indication, "Interview Record OK" on the 
paper recording the substance of the interview along with the date and the examiner's initials. 
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DESCRIPTION 



HANDHELD PORTABLE DIGITAL 
GEOGRAPHIC DATA MANAGER 



Background of the Invention 
It is often necessary for insurance adjusters, geophysicists, construction 
workers, real estate developers, and others working in the field to gather and 
assimilate geographic, topographic, visual (Le. 9 photographic), and other information 

10 about a site. These sites may be in locations which are remote and/or have no readily 
available access to computers or even power sources. Currently, there is no handheld 
portable device for gathering and processing such information. 

Printed maps are available for viewing almost all features that have 
geographical locations. A limitation of printed maps is that a user must manually sort 

15 through the entire map in order to find the relevant geographical features. 

Electronic maps have recently become available to replace paper maps for some 
applications. A map database is stored in a memory storage device as a bit map or 
as vectors that point to a map character. 

Many electronic location determination systems are available or have been 

20 proposed to provide electronic location information to a user equipped with a 
location determination receiver. Ground-based location determination systems are 
well known and include systems that were developed primarily for communications, 
such as cellular telephone, FM broadcast, and AM broadcast. 

The Global Positioning System (GPS) is a global navigation system that 

25 enables the user to utilize signals broadcast from satellites in order to identify 
positions. GPS systems are well known and widely used to accurately locate specific 
positions. GPS systems, which utilize a network of satellites to identify locations on 
the ground, are widely used in automotive and nautical navigation, construction, 
mining, and farming, as well as in a variety of other applications. See, for example, 

30 U.S. Patent No. 5,528,248 and PCT Application No. WO 95/05686. Other satellite 
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positioning systems, such as the Global Orbiting Navigational System (GLONASS) 
are also known. Cartographers, utilities operators, wildlife managers, and others have 
used GPS technology in conjunction with Geographical Information Systems (GIS) 
to augment map making procedures and other field work, particularly in remote 
5 locations. In such applications, there has been an increasing use of computers to 
facilitate data storage and retrieval. 

Personal Digital Assistant (PDA) is a generic name for a handheld personal 
computing device having a volume in the range of about 200 to 1200 cubic 
centimeters. PDAs can have as much computing power as some desktop personal 

10 computers and have been used in a wide variety of applications, including mapping. 

Unfortunately, hardware and software hurdles have limited the ability to utilize 
multicomponent data gathering systems in the field. Connecting equipment such as 
a GPS, a two-way radio, and a handheld pen computer for use in the field poses a 
number of obstacles. Many wires and cables work fine on a desktop computer but 

15 not on a handheld device. The essence of the subject invention is the development 
of a rugged, fully-integrated, easy to use handheld multicomponent field data 
gathering system for gathering positional, image, and other data, and simultaneously 
processing this information. 

20 Brief Summary of the Invention 

The subject invention concerns methods and devices which aid in collecting 
and verifying planning and engineering field data. The devices of the subject 
invention are handheld portable personal digital assistants (PDAs). The PDAs of the 
subject invention are designed to meet the needs of, for example, the construction 

25 and utility industry. The methods and devices of the subject invention are also 
applicable to other tasks such as environmental sampling, agricultural field data 
collection, property appraisal, and construction inspection. Specific applications 
include the gas, water, sewer, telephone, and cable TV utilities; in local, state, and 
federal government inspection agencies; and in the insurance and transportation 

30 industries. 
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The PDAs of the subject invention comprise multiple components interfaced 
with a central processing unit (CPU). Preferably, these multiple components include 
a digital camera, a GPS processing component, wireless radio communication 
capability, a digital compass with inclinometer, and user interaction capability. In a 
5 preferred embodiment of the subject invention, the PDA comprises at least two 
digital cameras which facilitate the determination of the distance from the PDA to 
an object. The digital compass and inclinometer of the PDA enable the 
determination of the orientation of the camera so that the exact location of an object 
which appears in the images from the digital cameras can be readily determined. 
10 In a preferred embodiment of the subject invention, the PDA comprises 

capability for wireless transmission of data to a host computer at a remote location. 
The PDA of the subject invention can further comprise a touch screen controller 
and/or a voice controller. The PDA of the subject invention further comprises a 

r 

power source. This power source may be, for example, a battery pack and any 
15 necessary electrical components to modulate the current so that it is compatible with 

the requirements of the various data gathering and processing components. 

In a preferred embodiment, the integrated handheld field digital data mapping 
device of the subject invention is capable of: 

(1) collecting and mapping global position system (GPS) data; 
20 (2) reporting GPS location, speed, and direction in a seamless fashion; 

(3) reporting orientation including magnetic direction, pitch, and roll; 

(4) collecting multiple pairs of color digital images (photographs) used to 
measure distance to an object; 

(5) voice command and voice query; 
25 (6) touch sensitive icon control; 

(7) providing icon-driven notes relating to field observations; 

(8) two-way wireless data transfer with host database station; 

(9) mapping the present environment in points, lines, and area features; 
and 

30 (10) providing navigation assistance to a desired location. 
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Brief Description of the Drawings 
Figure 1 is a block diagram showing the components of one embodiment of 
the device of the subject invention. 

Figure 2 is a schematic showing the determination of the distance to an object 
5 from the device of the subject invention using the images produced by two digital 
cameras. 

Figure 3 is a schematic of the embedded controller processor system. 

Figure 4 is a schematic of the digital compass and inclinometer. 

Figure 5 is a schematic of the flash memory subsystem. 
10 Figure 6 is a schematic of the GPS receiver subsystem. 

Figure 7 is a schematic of the LCD display subsystem. 

Figure 8 is a schematic of the touch screen interface and module. 

Figure 9 is a schematic of the stereo digital camera subsystem. 

Figure 10 is a schematic of the digital stereo range finder subsystem. 
15 Figure 11 is a schematic of the voice command subsystem. 

Figure 12 is a schematic of the wireless network interface. 

Detailed Disclosure of the Invention 
The subject invention provides materials and methods which are highly 
20 advantageous in collecting and utilizing field data for use in a wide variety of 
applications. The personal digital assistant (PDA) of the subject invention is a multi- 
component system for gathering positional, image, and other data, and simultaneously 
processing this information. The subject invention is also applicable to other tasks 
such as construction and utility monitoring, environmental sampling, agricultural field 
25 data collection, and property appraisal. Specific applications include the gas, water, 
sewer, telephone, and cable TV utilities; in local, state, and federal government 
inspection agencies; and in the insurance and transportation industries. 

The handheld data gathering device of the subject invention can, in a 
preferred embodiment, comprise: 
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(a) An imbedded computer capable of processing image data and 
interfacing to multiple data sensing modules. The user sees the 
graphical and alpha numeric results on a LCD touch screen, either 
color or black and white. The user controls the system through 

5 touching action icons or via voice commands. Alternatively, a mouse 

and keyboard may be used for more conventional user input and 
control of the system 

(b) A global position receiver (GPS) capable of accurately capturing 
geographic location with post processing or real-time differential 

10 correction capability. Better than one meter accuracy can be achieved 

with such a system. An interface from the GPS to the central computer 
to process the location data and the necessary software to display and 
record the results. 

(c) A digital compass capable of providing accurate heading (about one 
15 degree). An interface from the digital compass to the central 

processor and software capable of processing and displaying and 
storing the direction data. 

(d) Digital inclinometers capable of providing accurate pitch and roll data 
(less than one degree of error) for a plane parallel to the center line 

20 of the camera's field of view. The serial interface from the 

inclinometer to the imbedded computer transfers the pitch and roll 
data. The system software is responsible for processing, displaying and 
storing the pitch and roll data. It also makes the data available for use 
in calculating new coordinates. 

25 (e) A pair of digital cameras with matched optics mounted with the lenses 

in the same plane and with the line through the center of view of each 
lens parallel to the line through the other. An interface from the 
cameras to the imbedded computer processor and software capable of 
displaying, processing, and storing the images. 
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(f) Stereo image processing software or hardware algorithms capable of 
locating a selected object in one image in the stereo image pair. Once 
the object is located in the stereo pair image the displacement between 
the two objects is used in conjunction with the optical characteristics 

5 of the lenses and the spacing of the lenses to calculate distance to the 

object of interest. Accuracy of plus or minus one inch is achieved at 
fifteen feet, and about plus or minus a foot at thirty feet can be readily 
achieved using lenses which are only about nine inches apart. The 
distance accuracy drops off as the distance increases; however, longer 

10 range accuracy is achieved by sliding the cameras out to a wider 

spacing and/or attaching telephoto lenses to the cameras. For example, 
a 2x telephoto lens makes the accuracy plus or minus one inch at 30 
feet. The PDA can be made with lenses spaced at nine inches to keep 
the handheld unit small; however, when extended to 18 inches the 

15 accuracy is increased. The PDA can have a horizontal resolution of 

496 pixels. Image sensors with a higher resolution also increase distance 
accuracy. For maximum distance accuracy a higher resolution sensor 
is used in conjunction with widely spaced lenses and telephoto lenses 

(g) Data stored in the system may be sent to a base unit via a wireless 
20 network made up of two-way digital packet radios. Alternatively, data 

can be exchanged via an infrared link, a serial cable, a wired network 
connection, a PC Card, or a floppy disk. 
In a preferred embodiment, the PDA of the subject invention utilizes a pair 
of digital cameras which are not only able to provide photographic images of sites of 
25 interest but also to facilitate the determination of the distance from the PDA to any 
object in the images produced by the digital cameras. The distance determination 
is carried out by the PDA utilizing a software program which compares the images 
produced by the two cameras in order to determine the distance apart a single object 
is in the two images. This distance apart in the images is directly proportional to the 
30 distance that the object is from the PDA. Through a series of straightforward 

C:\DOOAP P\UF\UF-169P.DOC/DNB/mjc 



o o 

7 UF-169P 

computational steps, the PDA of the subject invention analyzes the distance between 
the object's images and converts this to the distance from the PDA to the object. 
The PDA simultaneously records the pitch and roll of the PDA as determined by an 
inclinometer, and it records the magnetic direction of the PDA as determined by a 
5 digital compass. This information, together with the GPS information, are all 
processed essentially simultaneously by the CPU and can be transmitted to a remote 
host computer by wireless communication components. 

The PDA of the subject invention further comprises a display screen and a 
means for the operation to interact with the PDA. This means of interaction may be, 

10 for example, by a touch screen or by voice recognition. 

A person skilled in this art having the benefit of the instant disclosure would 
readily appreciate that a variety of modifications could be made to the specific 
embodiments exemplified herein without departing from the spirit of the instant 
invention. For example, the GPS function of the PDA is for the purpose of providing 

15 a location for the PDA. This function could be performed by other locating means 

including, but not limited to, other satellite positioning systems and ground-based 
systems. Certainly, when the device of the subject invention is to be used indoors or 
in the vicinity of tall buildings, it may be advantageous to use a ground-based location 
system rather than a satellite system. Also, the device could display a map which the 

20 user could touch or in some other way designate the location of the PDA. In yet 
another embodiment, the PDA may be stationary, in which case its position will be 
known. 

Another embodiment of the subject invention utilizes more than two cameras 
in order to provide enhanced spatial determination of an object. In another 
25 embodiment, the cameras of the subject invention can be moved to increase the 
distance between the lenses so as to enhance the accuracy of distance determinations, 
especially for objects located at a substantial distance from the PDA. The moveable 
camera lenses can be retractable to their original position for convenience. The 
camera may utilize wide-angle lenses for objects close to the PDA. 

C:\DOC\APP\UF\UF-169P.DOC/DNB/mjc 
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In a further embodiment, the camera may include an infrared sensor for heat 
detection. The PDA may be equipped with additional chemical and environmental 
sensors to augment the type and amount of data which can be gathered. 

In a preferred embodiment, the PDA of the subject invention will have a real- 
5 time operating system to coordinate the essentially simultaneous collection of multiple 
types of data. 

The connections and interfaces of the various components of the PDA of the 
subject invention can be designed and produced by one skilled in this art having the 
benefit of the teachings provided herein. 

10 

Following are examples which illustrate procedures for practicing the 
invention. These examples should not be construed as limiting. All percentages are 
by weight and all solvent mixture proportions are by volume unless otherwise noted. 

15 Example 1 — Handheld Portable Data Manager 

In a specific embodiment, the PDA of the subject invention can run _ 
Microware's OS-9 (OS-9 Version 3.0 Documentation, 1995 Microware Systems Corp., 
Des Moines, Iowa) embedded operating system from ROM. The heart of the 
computer is a Gespac SBSG8-10 (Gespac Inc., Mesa, AZ) with two megabytes of 

20 ROM and one megabyte of static RAM. It has a Motorola 68340 CMOS low power 
processor with six serial ports. See Figure 3. 

A small carrier board holds two industry pack modules. The first has a 
GreenSprings IP-LCD VGA (Greenspring Computing Inc., Menlo Park, CA) 
controller interfaced with a small Sharp 640 x 480 TFT active matrix flat panel color 

25 screen. The color screen displays digital images, maps, and other data for verification 
and collection. See Figure 7. The second industry pack module is a storage module 
with eigh! megabytes of flash memory. See Figure 5. 

The user interacts via a Dynapro touch screen (Dynapro SC3 Touch Screen 
Controller and Software Users Reference Version 1.1, 1995, Dynapro Thin Film 

30 Products) on the sixth serial port and/or via voice prompting and command provided 
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by a OKI semiconductor voice control module connected to the first serial port. See 
Figures 8 and 11. 

A stereo pair of Gator Digital cameras (Gator Digital Camera Users Guide 
and Reference, 1994, Dycam, Chatworth, CA) are attached to the second serial port 
5 via a high speed synchronous connection. The two cameras provide stereo pairs of 

images for depth-of-field calculations and three-dimensional digitizing of objects of 
interest. See Figures 9 and 10. 

An eight-channel Trimble GPS receiver (Trimble Navigation, Sunnyvale, CA) 
is on the third port. A DCI differential correction receiver which corrects the GPS 
10 data to 1-2 meter accuracy is connected directly to the GPS receiver. See Figure 6. 

A Precision Navigation digital compass unit provides direction, pitch, and roll, 
as well as temperature, on the fourth port (Precision Navigation TCM2 Electronic 
Compass, 1994, Precision Navigation, Mountain View, CA). See Figure 4. 

The last port is used for communication using a Digital Wireless spread 
15 spectrum which communicates at distances of up to one mile line-of-sight. The 

wireless packet radio link uses the tcp/ip protocol for compatibility with sending and 
receiving data over the Internet. See Figure 12. 

The graphical user interface can be based on G-Windows (G- Windows and G- 
View Developers Manual, 1995, Gespac S.A., Geneva). G- Windows can run 
20 completely from ROM and provides an effective interface. G-View, a widget builder, 
is used for constructing object-oriented widgets to control the data sensors. The 
touch-sensitive color screen is used to display color and black-and-white digital images 
of objects under study and is used in the object documentation process. Objects are 
digitized by translating the GPS location of the data mapper to the object. The 
25 distance is calculated from the user touching the object of interest in the digital image 
and the computer automatically finding the same pattern in the stereo pair image. 
Bearing and orientation are calculated from the compass and inclinometer reading. 

When in voice command and control mode, the system verbally prompts the 
30 operator for input and flashes the appropriate icon when the voice command from 
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the user is processed. This leaves the operator's hand free for other tasks. The voice 
command and control also aids in poor visibility situations and for users with visual 
handicaps. 

5 Example 2 — Measurement of Remote Objects 

Objects of interest can be measured in the field by touching two points on one 
picture and letting the system calculate a three dimensional coordinate for each point 
from the direction, pitch, and roll data , lens characteristics and lens spacing. Distance 
between points can be calculated. All of the data is stored with the stereo pair of 
10 images for future processing with similar software on a desktop computer and/or if 
the user desires to capture more detailed information from the photo. 

Example 3 — Three Dimensional Digitizing of Objects 

As the user points the data collector unit in the direction of an object of 
15 interest the GPS is providing location coordinates, the system is also monitoring the 
direction pitch and roll of the stereo pair of cameras. Once the relative three 
dimensional coordinate for the object under study is calculated from the stereo data 
and the three dimensional heading of the camera the actual location of the object is 
calculated and recorded. 

20 

Example 4 — Mapping Objects in the Field 

The coordinate derived for the objects located in the stereo pairs of images 
is used to construct a geographic information system map layer. Each object can 
have attribute data stored with it including a photo of the object, its dimensions, and 
25 other attribute data the user desires to include. For example a verbal description can 
be digitized by the voice processor for a tag attribute or an icon driven system can 
be used to capture field observations. 
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Example 5 — Special Processing of Pictures 

Each stereo pair of pictures has location and directional data stored with it 
thus the plane of the photograph is known. This makes it possible to correct the 
picture for orientation. For example if the picture was taken at an angle the image 
5 can be rotated to remove the tilt in the picture. Also a floating scale is placed on the 
surface of an object in a picture so dimensions can be directly read off by the user. 
The units on the scale are set automatically set from the information obtained from 
processing the stereo images and their associated data. 

10 Example 6 — Field Correction of Existing Geographic Information S ystem Data 

Existing data is corrected in the field by loading an existing map data set on 
to the system. The user takes the map into the field. As features on the map are 
photographed the user touches the object on the map and the same one in the stereo 
pair of pictures. The system automatically calculates the displacement to the object 

15 and translates the GPS coordinate of the handheld data mapper to the object. This 
new accurate coordinate is then used to update the old feature coordinates including 
addend a altitude coordinate. 

Example 7 — Generation of Three Dimensional Wire Frame Sketches 
20 Several algorithms are used to enhance the edges of objects of interest. For 

example the lapse transform of an image will highlight the edges of an object in the 
picture. These algorithms are used to assist the calculation of the displacement 
measurement of the image pairs of the same objects in the stereo pair of digital 
photographs. Once the images are highlighted the system can then generate a wire 
25 frame three dimensional model for the object under study. In complex situations the 
user could take several pairs of images from different sides of the object under study. 
A more complete digital three dimensional map and model of the object is then 
constructed from the data. 
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It should be understood that the examples and embodiments described herein 
are for illustrative purposes only and that various modifications or changes in light 
thereof will be suggested to persons skilled in the art and are to be included within 
the spirit and purview of this application and the scope of the appended claims. 
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1. A portable, handheld data gathering device comprising a processing unit 
which is connected to, and communicates with, components which perform the 
following functions: 

(a) identify three-dimensional position of said device; 

(b) provide pitch and roll of said device; 

(c) provide digital photographic image; 

(d) identify direction of said image relative to said device; and 

(e) enable user interaction. 

2. The device, according to claim 1, which further comprises a means for 
wireless communication with a remote host computer. 

3. The device, according to claim 1, which further comprises a power source. 

4. The device, according to claim 1, wherein said three-dimensional position 
is identified using the GPS. 

5. The device, according to claim 1, which comprises two digital cameras. 

6. The device, according to claim 5, wherein the distance between the two 
digital cameras can be increased or decreased. 

7. The device, according to claim 1, which comprises an inclinometer. 

8. The device, according to claim 1, which comprises voice recognition 
capabilities. 
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Abstract of the Disclosure 
The subject invention provides devices and methods for efficiently and 
accurately gathering image and other field data using a handheld portable personal 
data assistant. 
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